1. Field of the Invention
The present invention generally relates to systems for monitoring and assessing human performance, and more particularly to use of biometric technology for determining the condition of an operator performing a critical task.
2. Background Description
Persons performing critical tasks may become incapacitated due to fatigue, inattention, stress, anxiety, alcohol, drugs, environmental or medical conditions. Resulting human errors or oversights may cause significant security and safety risks, or may decrease the quality of the work performed. It is therefore essential that the condition of critical personnel be monitored to provide evaluation of the quality of human performance and early warning of reduced effectiveness which might lead to human error. The International Standards Organization has established standards and certification procedures under which facilities, processes, and products are deemed to meet various quality levels. Condition monitoring and assessment of human operators can provide a similar technique for quality evaluation of the human labor component in certified processes.
Biometric identification techniques have been developed to insure that only the correct authorized person can obtain access to physical areas, computer networks, private information, valuable goods and services. However, most biometrics do not ensure that the subject is alive and able to perform the intended activity. Through biometric identity theft, an unauthorized person can use the fingerprint, iris pattern, voice print, or handwritten signature of an authorized person to gain access. The authorized person may no longer be alive when his biometric data is used, or he may be forced to cooperate against his will.
Drowsy, inattentive, and enraged drivers are major causes of accidents. Especially when those accidents involve trucks, they are often fatal and involve multiple vehicles. Similar problems occur for drivers of automobiles, trains, buses, and planes; with the primary causes being: sleepiness, stress, distractions due to cell phone use, distractions associated with tuning radios, eating, adjusting seat belts, talking or attending to passengers, medical emergencies such as stroke, seizure, or heart attacks, effects of medication such as cold and flu over the counter items. Among young drivers the problems are even more prevalent. Most teens admit they speed when driving. A majority don't wear seat belts. Nearly half teen drivers admit to sometimes being intoxicated or drug impaired. The drunk driver and the drowsy driver have been found to display the same type of inattentive behavior—with the same results.
A body of statistical evidence exists linking accidents to fatigue and inattention, but no such evidence has been compiled linking accidents to “road rage”. The general consensus is that such incidents are on the increase. While there has been no study on observables which would indicate an enraged driver, the assumption is made here that physiological measures such as blood pressure and temperature changes, agitated movements of the head, eyes, and hands, and erratic maneuvers of the vehicle would be correlated with rage. Those same observables could indicate other medical events as well.
Fatigue, inattention, stress, and rage are significant factors in the performance of other tasks besides driving. Computer operators, security watchmen, medical operations center personnel, utility and transportation control center staff, air traffic controllers, border patrols, soldiers, casino surveillance staff, surgeons, and others doing lengthly, high stress, or repetitive and boring tasks are prone to condition-related errors which can have significant security, safety, or economic results.
Drowziness, inattention, anxiety and stress are the cause of more than 50% of accidents and errors in the workplace. Alcohol and drugs, including over-the-counter, prescription, and illicit substances, are involved in more than 15% of detected incidents. Medical emergencies such as stroke, seizure, or heart attacks, are also serious concerns in persons performing critical or life-threatening tasks.
The problems are most acute in jobs involving shift work, high stress, physically demanding, or repetitive and boring tasks. Those characteristics describe many of the most critical jobs associated with security and safety: air and ground traffic controllers, border patrols, pilots and commercial vehicle drivers, and emergency response workers who alternate between periods of inactivity and periods of hyperactivity. Fatigue, stress, and the other conditions listed account for lowered physical and cognitive readiness to perform by persons who are in motion, such as patrol officers, firefighters, Hazmat response forces, soldiers, trauma response teams, surgeons, and SWAT teams. This range of conditions also accounts for lowered proficiency and productiveness, and increased errors in many types of tasks performed by seated persons attending to one or more displays such as at: security operations centers, power plant operation centers, 911 response centers, call centers, help desks, medical ICU desks, computer network activity monitoring desks, stock brokers, loan officers, etc. There are also important military uses. The Office of the Secretary of Defense has initiated a program on “Cognitive Readiness” which aims to assess the level of competence of soldiers in combat or personnel performing critical tasks. The IR-CMS system can be configured for fixed, mobile, or wearable use to accommodate all subjects.
Anticipating, detecting, responding to, and protecting against human accidents and errors requires the development of systems for realtime condition monitoring and assessment of essential personnel or any person whose condition or actions may place themselves or others at risk.
Physiological monitors used to provide realtime indications of fatigue, stress, anxiety, and medical condition generally require placement of electrodes or other sensors on the skin to produce Electroencephalograms (EEG), Electrooculograms (EOG), Electromyograms (EMG), Electrocardiographs (ECG), and measures of Respiratory effort, Respiratory airflow, and Oxygen saturation of arterial blood. In certain potential condition monitoring applications, such as for surgical patients, the subject is already strapped into a fixed position and contact sensors may be readily used. However, contact sensor approaches are impractical for continuous monitoring of most individuals at work.
The most extensive studies of condition monitoring for non-medical patients have involved fatigue detection in drivers. Three approaches to collecting and analyzing sensor data have each received significant attention and are the subject of numerous patents.
(1) Analysis of Facial Video to determine the percentage of eyelid closure over a period of time (PERCLOS), head nod, facial expression, and evidence of muscle tone.
(2) Physiological Sensor data such as EEG, ECG, temperature, respiration, oxygen level, and indications of sleep apnea.
(3) Driver Performance Indicators based upon vehicle movements such as lane changes, brake and jerk, speed changes, and failure to signal.
The PERCLOS system for analysis of eyelid closure percentage is considered the most reliable non-contact indicator of fatigue. That technique has several limitations, including its reliance on carefully positioned illuminators to produce reflections from the eyes to locate the eye positions relative to the camera. These limitations are especially troubling when ambient lighting conditions vary, making it difficult to detect eyelid status.
Various other methods have been proposed which use visual cameras to watch the driver's face and determine blink rate and gaze angle. Most use IR illuminators during darkness conditions. If glasses are worn by the driver, there is often a problem seeing the eyes due to glare from the illuminator, headlights, or ambient lights. In that case, blink rates and eyelid closure cannot be seen, nor can gaze angle. An estimate of gaze angle may be made from the nose position. Other condition monitoring approaches applicable to persons performing critical tasks are summarized below.
Analysis of rapidly changing human conditions is required in monitoring for loss of consciousness in high performance aircraft crew. In that application, sensors are required to actuate an alarm system or automatic pilot system to assume control of the aircraft. The automated decision making system must take into account individual crew member's physiological responses and conditioning. Previously, such monitoring involved anatomically invasive instrumentation devices or use of dermal sensing electrodes. Such devices are considered to be physiologically and psychologically discomforting to aircrew members.
Tripp (H1,039i, 1992) uses non-invasive sensing of arterial blood supply in the portions of the pilot's body adjacent to the cranium through the use of pulsating vascular bed optical signal transmission to perform intrusion-free physiological condition monitoring of pilots. Use of the physiological monitoring signals to generate alarm or assume control of the aircraft is also disclosed along with representative data associated with the sensed pilot physiological well-being indicators. While considered by Tripp to be non-invasive, his technique requires customized headgear or facegear which incorporates contact sensors. His only monitored condition is the cranial blood flow, which he asserts to be a generalized indicator of well-being.
U.S. Pat. No. 6,313,749 to Horne et al. uses driver or operator interrogation and response, combined with various objective sensory inputs on vehicle condition and driver control action, and translates these inputs into weighing factors to adjust a biological activity circadian rhythm reference model, in order to provide an audio-visual sleepiness warning indication. This approach does not provide the automated assessment of multiple indicators provided by the IR-CMS approach.
U.S. Pat. No. 6,107,922 to Bryuzgin uses a self-contained head set based sleep or fatigue alarm for a driver of a vehicle. It contains a set of arms extending from the alarm housing around the driver's head to the area under the driver's lower jaw. Involuntary relaxation of the driver's lower jaw causes rotation of the arms which in turn urges the movable contact against the stationary contact and therefore leads to the completion of the electrical circuit. After a predetermined delay, a buzzer or vibrator is activated to awaken the driver. This approach is considered highly invasive, and does not provide condition monitoring assessments.
U.S. Pat. No. 4,602,247 to Seko automatically determines driver rest periods and estimates when the driver might be fatigued. He does not perform direct condition monitoring of the driver and does not take personal variations or scenarios into account.
U.S. Pat. No. 4,502,122 to Yanagishima similarly estimates fatigue based upon driving time and driving conditions and then sounds an alarm.
U.S. Pat. No. 5,900,819 to Kyrtsos determines axle acceleration which exceeds a threshold, and from that estimates that the driver has jerked the wheel because he is fighting to stay awake. This approach has limited utility because it works only in the case where the driver displays such behavior. It does not detect early stages of fatigue, does not detect other conditions, and has no application to non-driver condition monitoring.
U.S. Pat. No. 5,786,765 to Kumakura determines the driver's normal blink rate at the start of a driving period and then monitors changes in the number of blinks per period. The driver is assessed as being drowsy if the number of slow blinks exceeds a threshold number. This is a version of the PERCLOS technique which uses visual face imagery only.
U.S. Pat. No. 5,689,241 to Clarke, Sr. et al. uses two infrared imagers to look for two indicators of driver drowsiness: one determines whether the eyes are open or shut, the second looks for temperature changes of the exhaled gas plume from normal breathing patterns. Clarke asserts that the gas plumes will lower in volume as the driver begins to hypoventilate, thus increasing the driver's blood level of carbon dioxide. Clarke states this is in most part the reason for early drowsiness associated with sleep. The combination of shut eyes and a decrease in breath temperature, which is a physiological response to hypoventilation thus initiating drowsiness, will trigger the infrared camera to zoom onto the eye region of the driver to watch for the eyes to open. This combined data is routed to the sleep status microprocessor memory via an optical image detector and thermal sensor for data changes above or below baseline data measurements. While Clarke uses IR sensors to monitor the face of a driver, he looks only at the temperature of the driver's exhaled breath and not at the face temperature. This approach would be affected by temperature variations due to air conditioning or heat from the vehicle, open windows, other passengers, and ambient changes in temperature. Since the temperature changes used to determine drowsiness are small, they can easily be overwhelmed by variations due to these other causes. Thus, the primary indicator of drowsiness is the detection of closed eyes, which is not an early indicator of impending fatigue. Clarke's approach does not provide a sufficiently wide range of condition monitoring indicators to be reliable.
U.S. Pat. No. 5,585,785 to Gwin et al. monitors the driver's hand grip pressure on the steering wheel, and sounds an alarm when the pressure falls below a lower limit. His technique does not use any imaging sensor, and provides no condition monitoring assessment.
U.S. Pat. No. 5,507,291 to Stirbl et al. describes a method and an associated apparatus for remotely determining information as to a person's emotional state. Stirbl uses active radiation beamed at an individual to elicit and measure reflective energy. He analyzes the differences in reflective energy and correlates that with emotional and medical states. In particular, he discusses polygraph applications. Stirbl is limited to the use of calibrated and known active radiation, aimed at particular locations on the body, and at particular angles of incidence.
U.S. Pat. No. 4,555,697 to Thackrey uses a teeth-held tilt alarm to signal that the driver has become drowsy. This approach is considered highly invasive and with few acceptable applications.
Therefore there is a need for a more reliable and robust condition monitoring system which is non-invasive and usable in a wide variety of situations where a human operator is performing critical tasks.